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ABSTRACT 

This publication provides vocational evaluators with 
guidelines to help facilitate selection and purchase of work samples. 
It describes seme of the essential elements of a well-constructed or 
"model" work sample (as defined in Vocational Evaluation and work 
Adjustment Association ( VEWAA ]-Commission on Accreditation of 
Rehabilitation Facilities [CARF] Vocational Evaluation and work 
Adjustment Standards with Interpretive Guidelines and VEWIA Glossary, 
Seven elements of the model work, sample are discussed: (1) 
apprcpriateness to client population, (2) work sample purpose, (3) 
relationship to available labor/training market, (4) work sample 
orientation (job description and test information! , (b) 
administration, (6) scoring, - and (7) learning assessment. The 
appendix provides a checklist suitable for modification which can be 
used as a ref€rence when evaluating a work sample. (i:LB) 



♦******♦****♦♦*♦♦♦***♦♦♦*♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦*♦♦♦♦♦♦♦♦♦♦♦♦♦♦*♦♦♦♦♦********* 

♦ Reproductions supplied by EDRS are the best that can be made ♦ 

* from the original document. * 
**♦*♦************♦♦*♦♦♦*♦♦*♦♦♦♦♦♦♦♦♦♦♦**♦♦*♦♦*♦♦♦*♦♦♦♦♦♦*♦♦***♦******''* 



SUGGESTED GUIDELINES 
FOR 

EVALUATING WORK SAMPLES 



BY 



Paul McCray 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



INFORMATION CENTER (ERIC)." 



U.S. DEPARTMENT OF HEALTH. 
EDUCATION & WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

THIS DOCUMENT HAS BEEN REPRO- 
DUCED EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION ORIGIN- 
ATING IT POINTS OF VIEW OR OPINIONS 
STATED DO NOT NECESSARILY REPRE- 
SENT OFFICIAL NATIONAL INSTITUTE OF 
EDUCATION POSITION OR POLICY 




September 1980 



TO THE EDUCATIONAL RESOURCES 



ADDITIONAL COPIES OF THIS PUBLICATION 
MAY BE PURCHASED FROM MDC FOR $2.00 PER COPY 




Paul R. Hoffman, Ed.D. 
Director, Materials Development Center 



Ronald R. Fry, M.S. Karl F. Botterbusch, Ph.D. 

Coordinator, Information Service Coordinator, Development Section 



Organizations conducting projects under government sponsorships are encouraged 
to express their own judgments. The points of view expressed in this pub- 
lication are those of the Materials Development Center and do not necessarily 
represent the opinions or policy of the National Institute of Handicapped 
Research- 



TABLE OF CONTENTS 

Page 

FOREWORD ^ 

ELEMENTS OF THE MODEL WORK SAMPLE i 

- Appropriateness to Client Population 2 

- Work Sample Purpose 

- Relationship to Available Labor /Training Market ... 6 

- Work Sample Orientation 13 

- Administration 15 

- Scoring ....... 18 

- Learning Assessment 1^ 

SUMMARY . . . - . . . / r . . . . . . . . 20 

REFERENCE LIST ^3 

APPENDIX 



ERIC 



4" 



FOREWORD 



Work samples are one of the most commonly used tools of vocational evalu- 
ation. Indeed, in many respects they often form the focal point for much of 
the assessment that takes place in many vocational evaluation programs. Be- 
cause of this, it is important that programs take special care to selectand 
utilize only those work samples that are accurate and reliable in measuring 
client interests, abilities, or aptitudes. Poorly constructed work samples 
can easily lead to erroneous, unsubstantiated observations and conclusions 
about client capabilities, and this eventually leads to dissatisfaction for 
both the client and the referral source. On the other hand, well constructed 
work samples are real assets in obtaining reliable and accurate estimates of 
client interests, aptitudes, and other job related factors. Thus, the ad- 
vantages of careful work sample selection with regard to meeting the general 
programmatic needs of a facility, and the specific, individual needs of 
clients are apparent. 

Vocational evaluators are frequently confronted with the difficult task 
of evaluating and selecting work samples for use within their programs. This 
process may take many forms. Evaluators may be interested in obtaining in- 
dividual commercial work samples or entire work sample systems. Perhaps they 
plan to independently develop work samples based on the local labor market. 
In other cases, primary interest is in obtaining noncommercial work samples 
developed by other programs or facilities. In addition to these examples, 
there are other instances where evaluators may not necessarily be interested 
in obtaining new work samples, rather primary concern is with reviewing the 
existing work samples already . in use within the facility in order to determine 
which merit continued, use, which should be upgraded, and which should be dis- 
carded; In all of these circumstances, the question that should be foremost 
in the evaluator's mind is how does one go about objectively assessing the 
quality and integrity of a given work sample? 

The Vocational Evaluation and Work Adjustment Association (VEWAA) and the 
"Commission on Accreditation of Rehabilitation Facilities (CARF) have worked 
together to formulate standards which spell out some of the essential 
characteristics of a work sample. 

3.4.3.1.1.7.2 If work samples are used: 

a. the vocational evaluation service work 
samples resources shall be representative 
of realistic competitive worker skills. 

b. work samples shall be established by an analysis 
of job tasks or traits related to a specific 
area of work,- and be standardized as to materials 

c. competitive norms or industrial standards shall 
be established and used. (CARF, 1980, p. 30) 



CARF has adopted these standards and now requires that they be applied as part 
of the vocational evaluation accreditation process. However, it should be 
understood that these standards only represent the most basic requirements 
regarding the structure and content of a work sample. As such, they are 
necessarily limited in scope. There are many other important work sample 
characteristics which evaluators should consider when critiquing a prospective 
work sample. 

The purpose of this publication is to describe some of the essential 
elements of a well constructed or "model" work sample. Few work samples 
presently available will contain all the elements described in this publica- 
tion. This should not, however, be considered as an indictment against a 
given work sample since it can't be assumed that all work samples, under all 
circumstances, must contain all of the elements of the model in order to be 
useful. Discretion must be used when comparing existing work samples to the 
model since many work samples are developed and utilized for highly specific 
reasons, and therefore, in some cases it is unrealistic to expect the inclu- 
sion of all elements. For example, a work sample developed for the sole 
purpose- of assessing range of motion or specific physical capacities need not 
necessarily have a client orientation which attempts to relate the task to 
specific occupations. Thus, the guidelines discussed in this oublication 
should be regarded as representing a flexible model which can be useful in 
attempting to systematically examine the content, utility, and design of 
individual work samples. 

It should be noted that these guidelines do not specif icajly apply to the 
many performance based tests designed to measure isolated aptitudes or traits. 
These' guidelines only apply to work samples as defined in VEWAA-CARF Vocational 
Evaluation and Work Adjustment Standards With Interpretive Guidelines and VEWAA 
Glossary (1978, p, 20J. Discussion of technical considerations such as work 
sample norms, reliability, and validity is limited to considering their re- 
lationship to CARF work sample standards (for further discussion of the stan- 
dards, refer to McCray, 1979b). The APPENDIX of this publication provides 
evaluators with a checklist which can be used as a reference when evaluating 
a work sample. Evaluators are encouraged to modify this form in order to 
better meet the particular needs of their own programs. 

It is hoped that after reading this publication, evaluators will: (1) have 
a more thorough understanding of the essential elements of a well constructed 
work sample, (2) be encouraged to critically review their program's existing 
work samples, and (3) plan to use this publication as *a reference when con- 
sidering new work samples for possible inclusion into their evaluation 
program. MDC would appreciate comments from practitioners concerning their 
reactions and suggestions for improving this document. In addition, evalu- 
ators who have established work sample review guidelines which they have 
found useful are invited to send them to MDC for consideration for possible 
distribution to the field. Full and proper credit is given to all con- 
tributions for materials used by MDC. 



Paul McCray, M.S. 
August, 1980 



ELEMENTS OF THE MODEL WORK SAMPLE 



In recent years, work sample testing has become a commonly accepted 
practice in vocational assessment. Its usefulness has spread beyond the 
confines of vocational evaluation and rehabilitation into other areas in- 
cluding: industrial psychology, personnel management, occupational therapy, 
and vocational education. There are many reasons for its increased acceptance 
as a viable assessment technique, the most common justification being that it 
has some unique advantages over the traditional paper-and-penci 1 testing 
procedures which have been employed for the last several decades. Sax (1973) 
has outlined a few of the major rationales for work Sc^mple testing. 

The major advantages for the work sample method include: (a) the 
work sample (developed by job analysis) is as close to the reality 
of work as we can get within the rehabilitation facility (except 
for actually putting clients to work in the facility), (b) it pro- 
vides exposure and experience on a wide range of jobs, (c) per- 
formance identical to work is required on the assumption that the 
closer the work sample is to the criterion, the more likely it is 
to be valid, (d) it not only assesses skills, but reveals aspects 
of the client's personality, interest, motivation, and attitudes 
towards the job, (e) clients respond more naturally to meaningful 
rather than abstract tasks, (f) it can eliminate cultural, educa- 
tional, and language barriers in the assessment of vocational 
potential, (g) many prospective employers are more receptive to 
utilization of work sample performance than predictors from other 
sources, (p. 32) 

Yet, in spite of these advantages, work samples are not without signifi- 
cant limitations. Sax (1973) went on to point out: 

Some of the disadvantages for the work sample method include: 
(a) work samples tend to emphasize quality and quantity of pro- 
duction rather than personality factors, (b) developing work 
samples for the many different types of job in the labor market 
is unfeasible, (c) workers are part of a working social family— 
and the social experience, reactions from co-workers, heat, 
noise, motivation, and wages vary so considerably in our shops 
that there is little comparison between the environment in in- 
dustry and the work .sample method, (d) weVe really not measuring 
the actual job, (e) because work technological change is so rapid, 
we run the risk of developing a good appraisal instrument for jobs 
which no longer exist, and (f) work samples have not often used 
statistical methods to develop reliability and validity information, 
(p. 32) 

Perhaps the most pragmatic attitude toward the use of work samples is 
represented by Nadolsky (1977) who suggests that vocational evaluators have 
a professional responsibility to critically examine work samples with regard 
to the iiistrument " s ability to realistically achieve the goals and objectives 
it sets forth for itself. The vast majority of individuals working within 
the field of human measurement will readily agree that there are "good" tests 
and "bad" tests. Buros (1977) has gone so far as to state "... most 
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standardized tests are poorly constructed, of questionable or unknown valid- 

i-ty-r Pi'Stentious in their claims, and likely to be misused more often than 

not" (p. 1972). Thus, the importance of being able to discriminate between 
useful tests -and those that may be poorly designed, is evident. Certainly 
there aremany ^work samples that have merit as well as some that do not. If 
this problem is recognized, evaluators should be able to avoid the trap of 
presuming that all or most work samples, due to their very existence, are 
necessarily useful tools. Evaluators must be prepared to critically examine 
every prospective work sample on an individual basis, so that v they will select, 
and utilize only those tools which will contribute to a valid and reliable 
understanding of a client's immediate and long-term performance capabilities. 
It is, therefore, imperative that vocational evaluators have a sound under- 
standing of the essential elements of a work sample. 



Appropriateness to Client Population 

Botterbusch (1976) has suggested that an important consideration in work 
sample selection is the appropriateness of a work sample's design with regard 
to corresponding to the abilities and limitations of the client population 
being served. Appropriateness takes into account many factors including: 
prerequisites, instructional methods, job demands, and task complexity. All 
of these factors combine to influence the potential utility a work sample 
will have with a given population of clients. Where a program is serving a 
wide variety of clients, it is likely that most work samples will be appro- 
priate for at least a small segment of the client population. However, in 
cases where the client population is relatively homogeneous (e.g., mentally 
retarded), it is imperative that evaluators understand the critical relation- 
ship between work sample demands and client capabilities. When these factors 
are not taken into account, it is easy to obtain work samples that cannot be 
used with the vast majority of clients simply because of instructional limi- 
tations or task complexity. 

One of the most fundamental considerations in work sample selection is 
whether the prerequisite skills required to perform the task are likely to be 
held by the majority of clients that will be asked to take the work sample. 
Most work samples require certain prerequisite skills such as ability to read, 
to hear, and to write. It is imperative then, that the majority of people for 
which the task is selected, will have the prerequisite skills. If this is not 
the case, much money and time may be invested in a tool which is largely use- 
less. For example, a program serving a retarded client population would 
generally be ill-advised to select any work samples requiring sophisticated 
reading skills or the ability to operate complex audiovisual equipment as a 
prerequisite to work sample performance. Similarly, a facility serving a 
vision-impaired population would generally not want to use work samples that 
require normal eyesight, unless of course, appropriate adaptations can be 
made for the tasks. 

The second important criteria is the relationship between the occupations 
the work sample purports to relate to and the anticipated level of client 
capabilities. This notion is indirectly supported by CARF Standard 3.4.3.1.1.6 
which states, "job areas assessed in vocational evaluation shall be based on 
client capability . . ." (CARF, p. 30). If a program is serving a client 
population which is largely composed of individuals with normal cognitive 
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skills but severe physical impairments, work samples which relate to jobs 
requiring unimpaired physical capabilities would usually be inappropriate, 
since the job areas assessed would not be realistic in view of client limi- 
tations. Certainly evaluators should avoid obtaining work samples based on 
jobs which are either too complex or too simple in relation to client capa- 
bilities. In many cases, this relationship can be determined by briefly 
examining the work sample, assuming the evaluator has a sound understanding 
of the strengths, limitations, and interests of the client population. 

Another important consideration is the instructional technique used during 
the work sample administration process. As much as possible, it should par- 
allel the method used in industry for similar jobs and also be compatible with 
client learning styles. For example, if a work sample simulates a routine, 
repetitive assembly operation and it is recognized that in industry workers 
are trained via an oral and demonstration instructional technique, then 
evaluators should look for work samples that use that technique. A work 
sample that uses a sophisticated written format may be beneficial in terms 
of reducing the amount of time the evaluator spends with the clients, but it 
may not accurately portray the actual job. In special circumstances where 
client learning styles are not compatible with the normal instructional format, 
e.g., severely retarded clients being given verbally loaded instructions, then 
the evaluator should study the prospective work sample to determine whether the 
Tristructional strategy could be adapted to better correspond to client learning 
styles. For example, the instructional format for a bench assembly task might 
be altered to a hands on, "try another way./' technique for a severely retarded 
client population. Although it may seem that any instructional technique can 
be readily adapted to client needs, this is not always the case. Instructions 
which are very well presented via an audiovisual format may not be easily con- 
verted to a written, hands on approach because of task complexity, evaluator 
time constraints, or need to show motion. Failure to insure that the instruc- 
tional mode parallels that of industry and is compatible with client learning 
styles can contribute to erroneous conclusions about client capabilities 
(McCray, 1979a). 



Work Sample Purpose 

Perhaps one of the most critical characteristics of the model work sample 
is that it have clearly stated purposes and objectives. In other words, the 
purpose(s) for which the work sample was developed must be concretely defined. 
Such a description must go beyond simply stating that "the work sample was 
developed to assess work potential or job interests." The specific occupa- 
tions, interest factors, aptitudes or abilities which the tool is designed to 
address should be spelled out in the work sample manual (e.g., "ability to 
operate an industrial drill press, interest in unskilled, gross assembly bench 
work tasks, upper body range of motion, mechanical aptitude"). Definitions of 
each of these factors should also be provided so that users will have a re- 
liable understanding of the specific factors which they are interested in 
measuring. 

The Durposes and objectives for which a work sample is designed should, 
at least on a broad level, relate with the goals of the vocational evalu- 
ation program. When a work sample is first being considered, evaluators 
should keep in mind how that specific tool will help better meet individual 
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client needs, which will in turn help meet program goals. If a work sample s 
Durpose is not clearly defined, it is difficult to estimate the extent to 
which the work sample corresponds to the goals of the program. For example, 
imagine that a vocational ' evaluation program's primary goal is to provide a 
basis for systematic occupational exploration to a client population primarily 
composed of school age students. In a case such as this, if the program wished 
to use work samples as part-of the exploration process, then primary emphasis 
should be Placed on obtain'ing work samples whose objectives are commensurate 
with those' of the program. Thus, work semples primarily designed for use as 
occupational exploration tools would generally be more useful than work samples 
that have questionable face validity and measure only an isolated trait. In a 
case like this, the isolated trait work sample would be a poor simulation of a 
job or job area and, therefore, the program objective of occupational explora- 
tion, would not be adequately met by a work sample designed for a different 
purpose. 

Another example which commonly occurs in vocational evaluation programs 
is the practice of using work sample testing as a basis for predicting the most 
approoriate job or training avenues for the client. For example, if a program 
defines its major goal as the accurate assessment of job potential , then it is 
imperative that the facility have on hand assessment tools and techniques which 
have substantive evidence that they too have similar goals and objectives, 
i.e., they can serve as a valid basis for job prediction. If in such a case 
an evaluator wishes.-to use work samples as a basis for making specific pre- 
dictions with regard to employabil i ty potential in certain jobs or job areas, 
it is Important that the evaluator carefully critique all the work samples 
being used with regard to their ability to help achieve the goal of job 
prediction. In some cases, this process will lead the evaluator to the 
conclusion that the actual purpose of the work sample was only marginally 
related to job prediction, e.g., interest or isolated trait assessment. In 
such cases, the evaluator must recognize that program goals are not in line 
jj/ith the stated work sample goals. Thus, the work sample is being used for 
'the wrong purpose. Evaluators would never think of using aptitude tests to 
assess interests, yet in many ways, this is what is being done when evalu- 
ators regularly use work samples for purposes for which they were not 
designed and which may, in fact, be different from program goals. 

Most vocational evaluation programs utilize more than one isolated as- 
sessment technique or tool. Indeed, CARF (1980) has stated in Standard 
3.4.3.1.1.7 that all vocational evaluation services shall use two or more 
of the following assessment techniques: work samples, psychometrics , situ- 
ational assessment, or job site evaluation. Therefore, when critiquing new 
work samples, evaluators must recognize when they already have on hand other 
work samples and evaluation techniques which serve essentially the same pur- 
poses as the prospective work sample. Obtaining several new work samples 
whose purposes duplicate those of other work samples, psychometri cs , job sites, 
etc., should generally be avoided as it is costly and can contribute to the 
development of a redundant assessment process wherein clients perform a series 
of tasks, all of which are essentially the same, although they may, on the 
surface, look different. For example, if a facility has set up a series of 
clerical job sites as evaluation tools, there would be little reason for an 
evaluator to obtain clerical work samples which would differ little from the 
job sites. Time and money can be better spent on increasing the variability 
and scope of the assessment tools' -purposes rather than duplicating what is 



already available sirnply because of convenience or failing to understand the 
specific rationales for the work samples being used. 

The addition Ih" a work sample to an existing work sample testing rorogram 
should be the result of having identified a concrete client-centered need which 
is not being met by the available work samples, but which could be satisfied 
by the prospective work sample. For example, if a new industry such as elec- 
tronics assembly moves into the area and it appears that clients could be 
placed in this occupational area, but no work samples or other assessment 
devices which assess ability to do this kind of work are on hand at the 
facility, then there would be a clearly established need for this tool. It 
could be a valuable asset to the program and would not duplicate the existing 
tools. Similarly, if a facili-ty-'had a series of mail sorting and filing work 
samples ^hich were useful, but still failed to measure certain isolated yet 
critical factors, e.g., ability to use a postage scale and read a weight/zone/ 
cost chart, then the selection of an additional mail sorting and filing work 
sample which specifically addressed this need could be very useful because 
the new work sample would compliment the existing tools. 

\ Work samples that' clearly define their purposes are advantageous in many 
respects. First, this information can be very useful for evaluation planning. 
For example, if a work sample has certain distinct purposes and the evaluator 
knows the purposes for which the client was referred, then it becomes a rela- 
tively easy task to match work sample purposes to evaluation objectives. Thus, 
if one of the purposes of evaluating a client is to assess upper body range of 
motion, then the evaluator should be able to readily identify;work samples 
which are designed to assess upper body range of motion. 

The second advantage is that work samples with clearly defined purposes 
make it easier for evaluators to determine the extent to which certain work 
samples overlap in purpose. This information reduces the frequency of con- 
tinuing to develop or purchase additional work samples which"**may superficially 
appear to be different but which are in fact, serving basically the same 
functions. If a program has several work samples, all of which clearly specify 
that their purpose is to assess eye-hand-foot coordination, manual dexterity, 
and fine finger dexterity, then it becomes apparent that there is no need for 
additional tools that measure the same factors. Yet, if these purposes are 
not clearly spelled out, different evaluators will often have different per- 
ceptions of the same work sample's purpose, and thus the temptation to obtain 
additional, albeit functionally equivalent work samples, increases. 

Work samples with clearly defined purposes are also useful when orienting 
a client to a work sample task and when counseling a client as to the results 
of the overall evaluation. Because the evaluator knows the specific objectives 
of the work sample as v/ell as the reasons why it was administered, this factor 
can be more reliably and frequently more meaningfully communicated to the 
client. It is often easier for clients to understand the rationale for a task 
and the significance of their performance if the evaluator can clearly in- 
dicate that "the purposes of work sample X were to assess ability to: sort 
by alphabet, sort by number, repair a. hydraulic brake, etc., and on those 
measures the results indicate . . 

Last, clearly defined work sample purposes facilitate communication 
between the evaluator and the referral source. It is often advantageous in 



a client staffing for the evaluator to be able to define the specific reasons 
for which a work sample was administered. By delineating the concrete purposes 
of the task, the evaluator is suggesting that work samples are given for 
specific reasons, to answer specific questions; and a routine, mechanical , 
"shotgun" approach wherein every work sample is administered simply because 
it is available, is avoided. This lends credibility and professionalism to 
the program and staff. In addition to this> through continued close associa- 
tion with the referring counselors, the counselors gain familiarity with the 
work samples available and their purposes and may, as part of the referral 
process, request that certain work samoles be administered. This procedure 
is paralleled in traditional psychological testing when referral sources 
request that specific paper and pencil tests be administered to a client. 



Relationship to Available Labor /Training Market 

Botterbusch. (1976) has pointed out that one of the basic foundations of 
an effective vocational evaluation program is a thorough and accurate under- 
standing of the potential employment/training opportunities realistically 
available in the communities from which each client is being served. Some 
vocational evaluation programs find themselves serving clients referred from 
communities with relatively narrow, homogeneous labor markets. Other programs 
serve a broad range of clients coming from a heterogenous or widely-varying 
labor market where the employment/training opportunities are seemingly un- 
limited. In either case, and because many work samples are developed directly 
from the Dictionary of Occupational Titles listings, it is imperative that 
prior to selecting assessment tools avaluators be aware of what types of 
opportunities actually exist within the client's available labor market. Once 
this is known, work samples can be reliably and objectively selected on the 
basis of their relationship to the needs of clients and the potential job 
market. Failure to recognize the unique limitations and opportunities that 
exist within any specific job market can contribute to the purchase and/or 
development of work samples which have limited practical utility, i.e., job 
predictions can be made based on work sample performance, but because jobs 
are not available in that occupational cluster, the assessment and its out- 
comes are largely irrelevant. When this happens, vocational evaluation can 
easily become a service of limited practical use. 

Recognizing that it is important to understand the available labor market, 
one must decide on how to go about accumulating this knowledge- If at all 
possible, it should be approached in a logical, systematic manner so that the 
information derived from this research can be combined into an organized 
picture of an area's labor demands and opportuni ties\ Once this profile is 
generated it should be periodically updated and maintained on file for future 
reference, since it may indicate significant trends, e.g., certain businesses 
such as electronics plants^have hired a large number of clien^,s over a one 
year period; an occupational cluster exists for which the facility has no 
work samples; or an occupational area that is heavily assessed during the 
evaluation process actually ha.^ very few, if any, job openings. The ad- 
vantage of accumulating and keeping this information in an organized manner 
is that rather than assuming 'one or two staff members are carrying it around 
in their heads, it is documented and available for the entire staff. Such 
information cannot only be useful to evaluation personnel, but also job 
developers, contract procurement people, salesmen, work adjustment staff. 



c 



6» 



job training, and production personnel. It also means that when n^w evalua- 
tion personnel come into the facility, accurate and timely job market in- 
formation will be available to the^i immediately, thereby, eliminating a time 
lag of' several weeks or months wherein new evaluators must build up their 
knowledge of the available labor market. It is also advantageous in that the 
information does not disappear when the one or two staff members that have 
accumulated it eventually leave the facility. 

Fortunately, there are some relatively easy ways to accumulate up-to-date 
labor market information. First, in order to get a general picture of the 
major industries in the areas where clients would typically seek employment or 
training, evaluators can contact the appropriate Chamber of Commerce which 
usually has the information available. In addition to this, many urban areas 
often have manufacturer's guides which describe many of the manufacturing and 
service industries in the area. Evaluators should also locate and identify 
the educational/vocational training programs that are available. Many times, 
business colleges, junior colleges, and vocational-technical schools provide 
relatively specific job training programs that would be appropriate for cli- 
ents. Thus, evaluators must be aware of the educational training programs 
that are also available. Once the evaluator has a general understanding of 
the broad spectrum of job opportunities, more detailed information related to 
the specific kinds of jobs existing within the aforementioned industries will 
be wanted. 

Another way for evaluator* s to accumulate specific job information is to 
work with the state employment service which has labor analysts and up-to-date 
listings of local as well as state and regional employment opportunities. This 
data will provide the evaluator with specific job listings and labor market 
predictions which can be studied in order to gain a relatively objective per- 
.spective of different communities' labor needs. Evaluators should also study 
the want ads of newspapers as well as contact private employment agencies in 
order to get a broad-based view of the labor market. 

Once an evaluator understands the broad occupational areas existing within 
the community as well as several of the specific jobs that exist within those 
industries, it is then possible to intelligently compare the extent to which 
the program's work samples and other evaluation tools/techniques correlate 
with the labor market. Occasionally this can be done by comparing the job 
titles listed in the work sample manual with those listed in the labor market 
survey. Although this is a useful method, it has one significant limitation; 
a local job may have the same or similar title as one listed in the work 
sample, but the two tasks, when examined closely, may actually be quite dif- 
ferent. How then, does an evaluator reliably determine the extent to which a 
work sample relates to specific jobs within the labor market?. One useful 
method is to perform a job analysis on the work sample and then compare that 
information to the job description provided in the Dictionary of Occupational 
Titles , the job description provided by the employer's personnel department, 
or even better, do a job analysis on the actual job and then compare that to 
the work sample job analysis. This will give highly specific information 
about the work sample's relationship to the" actual job market. 

Implementing some of these ideas will help evaluators better determine the 
extent to which the program's work samples represent the actual job demands of 
the available labor market. In many cases, evaluators will find gaps in their 
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work samDie orogram, i.e., viable occupational areas are not accurate y rep- 
resented' by the available work samples, not represented at all, or an mi tea 
number of occupational groupings are receiving most of the work sample 
development attention at t'le expense of other viable .lob areas. By doing 
this carefully and systematically, the evaluation program is establishing its 
credibility for referral sources with regard to having evidence that the 
program actually is a community-based resource for realistic, practical 
vocational assessment. 

It has already been suggested that job analysis is one method for deter- 
mining the reoresentativeness of a work sample. Work sample reoresentati veness 
refers to the' degree to which a work sample simulates the job(s) it purports to 
relate to. This concept is supported by CARF (1980) Standard 3.4.3.1.1.7 2a 
which states, "the vocational evaluation service work samples resources shall 
be representative of realistic competitive worker skills" (p. 30). 

Work sample manuals that provide concrete information regarding the work 
sample's representativeness are highly desirable because they allow the eval- 
uator a ready reference for determining, to some extent, the degree to which 
a work sample actually simulates the tasks, aptitudes, traits, etc., required 
of specific jobs. There are many ways the content validity of a work sample 
can be established. Dunn (1971) has outlined one practical format that es- 
sentially involves comparing DOT task listings to the work sample tasks. 



Fountain Server, DOT Code 319.474-010 


DOT Task Listing 


Work Sample Tasks 


Prepares soft drinks and ice cream 
dishes from memory or by following 
oral directions 


Prepares soft drinks and ice cream 
dishes by following oral directions 


Serves the above foods 


None 


Cleans and polishes glasses, 
dishes, and fountain work. 


Cleans and polishes glasses and 
dishes 


May prepare and serve sandwiches 


None 

. „..,. 



If this information is contained in the work sample manual, the evaluator 
can determine the degree to which the work sample represents the job of 
Fountain Server as described in the DOT . 

McCray (1979b) has suggested a similar method for work sample content 
validation . 
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Fountain Server, DC 


)T Code 319.474-010 


• Job Analysis Description 


Work Sample Tasks 


Prepares soft drinks and ice cream 
dishes from memory (70%) 

Cleans and polishes glasses, dishes 
and fountain work (155^) 

Handles money to make appropriate 
exchanges for ourchase of fountain - 
items. Operates NCR cash register 

(lO^o) 

Sweeps fountain area and removes 
appropriate rubbish (5%) 


Prepares soft drinks and ice cream 
dishes by following oral directions 
(90%) 

Cleans and polishes glasses and 
dishes (lO^o) 

None 
None 



Note that this particular format provides the work sample reviewer with 
more detailed information regarding the content validity of the work sample. 
The second example is based on an actual .lob analysis rather than a DOT des- 
cription, and indicates that the job actually differed significantly in terms 
of task requirements from the more general DOT description. The use of a job 
analysis provides a method whereby the percentage of time required on each task 
can be specified and, therefore, available for comparison. The end result is 
that the latter method often provides a more detailed picture of the repre- 
sentativeness of the work sample. 

The third type of content validity format that evaluators can look for 
when critiquing work samples involves the provision of a task matrix grid as 
evidence of content validity (a more complex work sample/ job task has been 
selected in order to better show this procedure). 



WORK SAMPLE TITLE 
Cashiering 

Related Jobs Ind. DOT No. 


Receives cash or checks 
from mock customers 


Issues appropriate change 
to mock customers 


Operates cash register 
to record transactions 


Completes credit charge 
transactions using charge 
plates 


Verifies balances in 
register aqainst receipt 
totals 


Totals cash and checks 
for bank deposit 


Fills out bank deposit 
slips 


TOTAL 


Cashier I clerical 211.362-010 


X 


X 


X 


X 


X 


X 


X 


7 


Cashier II clerical 211.462-010 


X 


X 


X 


X 


X 


X 


X 


7 


Supervisor - hotel & rest. 

LabmGi b iCL. Li . c. J. J. . i. ou — VJi.U 












X 


X 


2 


oupervisor - 

Food Checkers hotel & rest. 211.137-014 
and Cashiers 










X 


X 


X 


3 


Teller finan. inst. 211.362-018 


X 


X 








X 




3 


rayiTiaSLcr OT 

Purses amuse. & rec. 211.367-010 


X 


X 








X 




3 


Uabii 1 CI — 

Checker ret. tr. 211.462-014 


X 


X 


X 










3 


Cashier — 

Wrapper ret. tr. 211.462-018 


X 


X 


X 


X 


X 






5 


Cashier — 

Gambling amuse. & rec. 211.462-022 


X 




X 




X 






3 


Check 

Cashier bus. ser. 211.462-02t 


X 


X 


X 










3 


uri ver s - 

Cash Clerk motor trans. 211.462-030 


X 




X 




X 






3 


161 1 6r 1 i^riL^ rica L 

& power 211.462-034 


X 




X 










2 


Tol 1 

Collector gov. ser. 211.462-038 


X 


X 






X 






3 


Cashier - 

Courtesy Booth ret. tr. 211.467-010 


X 


X 


X 




X 






4 



WORK SAMPLE TITLE (Cont.) 
Cashiering 

Related Jobs Ind. DOT No. 


' Receives cash or checks 
, from mock customers 


Issues appropriate change 
to mock customer 


Operates cash register 
to record transactions 


Completes credit charge 
transactions using charge 
plates 


Verifies balances in 
register against receipt 
1 totals 


Totals cash and checks 
for bank deposits 


Fills out bank deposit 
slips 


TOTAL 


Parimutuel - 

Ticket Cashier amuse. & rec. 211.467-018 




X 


X 




X 






3 


Sheet Writer amuse. & rec. 211.467-026 




X 






X 






2 


Ticket Seller clerical 211.467-030 


X 


X 


X 




X 






4 


Cashier - 

Tube Room ret. tr. 211.482-010 


X 


X 






X 






3 


Food 

Checker hotel & rest. 211.482-014 






X 










1 


Food & Bever- 
age Checker hotel & rest. 211.482-018 






X 










1 



It is evident that there are certain advantages to this type of content 
validation procedure, the primary one being that the evaluator can readily 
discern all the jobs related to the work sample and the degree of overlap 
between work sample and job tasks. Yet, there are two significant limitations 
which evaluators must be aware of. First, the matrix does not depict related 
job tasks that are significant yet not addressed by the work sample. For ex- 
ample, the job of Cashier I appears to be thoroughly represented by the work 
sample; all the work sample tasks are required on the job. The evaluator may 
therefore quite easily assume that because of this, the work sample is neces- 
sarily fully representative of the job, cashier. Yet, this may not be the case, 
since the actual job of Cashier I may require other significant skills (e.g., 
alphabetically stocking shelves) which are not measured by the work sample. 
In other words, all of the tasks actually required on the job are not listed 
on the matrix. Only the tasks required of the work sample are listed. This 
problem emphasizes the importance of evaluators thoroughly understanding the 
individual work sample tasks as well as all of the significant tasks required 
by a given job. 

The second problem revolves around the fact that the relative times re- 
quired of each task are not spelled out. For example, the work sample task 
"operates cash register to record transactions" may occupy 20% of the time it 
takes to perform the work sample, yet on the job of Cashier I, which is said 
to require this activity, 75% of a worker's time might be devoted to this task. 
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This discrepancy could be imoortant when making decisions as to client capa- 
bilities, and certainly influences one's perception of the degree to which the 
work sample simulates the actual job. Yet with this particular type of format, 
such information is frequently not available. 

In any -lease, it is imperative that if a work sample is being used to pre- 
dict job success or measure specific job interest, it should be representative 
of the skills, aptitudes, and abilities needed to do the job. If it is not, 
client job interest may be based on an inaccurate depiction of the job and any 
predictions as to specific job placement potential may be at least partly 
incorrect. 

Finally, evaluators must recognize that although work samoles may be con- 
venient, they are not always a realistically viable assessment technique for 
many occuoations. In some cases, evaluators would be better advised to develoo 
other evaluation tools such as job sites or situational assessment programs in 
order to best evaluate a client's job potential. For example, many service oc- 
cupations such as cook, maid, or dishwasher would be difficult to quantify into 
a work sample; job site evaluation could provide a more effective basis for 
evaluation in these areas. Other more cognitively oriented occupations in the 
technical, managerial, or professional areas are often better assessed by job 
specific paper and pencil tests. Therefore, before evaluators go to the ex- 
pense of developing or purchasing a work sample, they must carefully consider: 

(1) can the work samole realistically assess what it purports to assess, 

(2) would other assessment techniques be feasible and provide a more effective 
basis for decision-making, and (3) to what extent is the work sample related to 
the available labor market? 

Face validity is another concept important to work sample testing; al- 
though, it generally receives minimal recognition in traditional psychological 
testing theory. Lyman (1978) has described face validity as "the least im- 
Dortant of the indications of validity" (p. 29). Anastasi (1968) has stated, 
'"fundamentally, the question of face validity concerns rapport and public 
relations . . . face validity itself is a desirable feature of tests. Cer- 
tainly if test content appears irrelevant, inappropriate, silly, or childish, 
the result will be poor cooperation, regardless of the actual validity of the 
test" (p. 104). 

Face validity may generally be thought of as the extent to which a test 
"looks" valid. It is an important concept with regard to vocational evalu- 
ation in general and work sample testing in particular since one of the most 
common rationales for using work samples rather than paper and pencil tests 
is that vocational evaluation is essentially a work assessment process. 
Therefore, the tests used should look like real work to clients and as Anastasi 
suggests, this should help minimize the likelihood of unreliable client per- 
formance. The concept of face validity is particularly important when one 
recognizes that many of the clients served in vocational evaluation have been 
low achievers in school and are likely to be discouraged by what appear to be 
the usual paper and pencil tests which have historically emphasized their 
inadequacies. Nadolsky (1977) has described face vaTidity as "... an 
essential feature of a work sample. It enables the individual who performs 
the work sample to identify its contents as being relevant to a particular 
occupational area" (p. 7). 



The extent to which a work sample looks like real work is strongly in- 
fluenced by the representativeness of the tool. For example, a job sample 
which by definition fully represents an entire job (with the possible exception 
of the overall industrial environment) is the most representative type of work 
sample possible and, therefore, will look very much like real work, i.e., it 
will have a great deal of face val idity. Other work samples, however, may look 
progressively less and less like a real job or even a work related task. For 
example, a work sample measuring an isolated trait such as eye-hand-foot 
coordination will usually have much less face validity than a job sample.. Yet, 
it may generally be said that the majority of single trait, or cluster trait 
work samples will still appear to clients to be more relevant to work than an 
abstract paper and pencil test. Thus, evaluators should be careful when re- 
viewing work samples to avoid those that claim to be work samples but actually 
are little more than another series of paper and pencil tests. 

When choosing between trait oriented work samples and a job sample which 
more closely resembles real work, evaluators must be sensitive to the specific 
need which the prospective work sample must meet and the extent to which they 
feel face validity will impact on their clients. Jf the general client popu- 
lation being served is comprised of individuals with only physical disabilities 
and relatively high level cognitive skills, their interest and motivation may 
be unaffected by work samples that have minimal face validity. On the other 
hand, if most of the clients have below average cognitive skills, evaluators 
may prefer work samples which have very obvious face validity since this 
factor may make it easier for the client to understand and accept the purposes 
of the evaluation (i.e.., to assess job potential and not "pass or fail'*) and 
this may facilitate client motivation and interest. 



Work Sample Orientation 

A work sample orientation essentially consists of providing each client 
with (1) job qualifications, salary, opportunities for advancement, training, 
working conditions and (2) test information"-purpose of the test, how it 
relates to work, and factors to be assessed. A concrete, organized orienta- 
tion to a given work sample is highly beneficial because it adds meaningfulness 
and relevance to the task, particularly if the evaluator can point out specific 
vocational opportunities within the local labor market which are directly re- 
lated to the work sample. Many times the purpose of a task is not readily 
apparent to a client although the task's objectives may seem obvious to the 
evaluator. A work sample which may only appear to be marginally related to 
a job can especially benefit from an orientation because explaining the purpose 
of the task to the client helps ensure that the client is less likely to fail 
to recognize the relationship between performance and potential job/training 
opportunities. 

A thorough work sample orientation also provides the client with a compre- 
hensive -understanding, of a job or job area. Although the physiqal makeup of 
the work samfile and 'performance of the task may give the client a basic under- 
standing of the skills and aptitudes required on a job, a client orientation 
provides additional information with regard to salary, fringe benefits, and 
advancement opportunities. This added information helps the client gain an 
accurate understanding of the entire job, not just the skills needed to 
perform the job. 
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Because a client orientation facilitates an accurate understanding of many 
diverse aspects of a job or job area, a client's job interests can be more 
reliably determined. In other words, a client's perception of a job is based 
on accurate, detailed information and any expressed interest is likely to be 
relatively stable since that interest is not based on an incomplete, partial 
picture of a job. For example, a client may express a stronci interest in a 
'job based on his knowledge of the skills involved, but when he is encouraged to 
also consider the -amount of training needed or the opportunities for advance- 
ment, he decides that he is no longer interested. The orientation, therefore, 
helps the evaluator and the client assess the latter's interests, attitudes, 
and:::knowledge of occupational information, which in turn is related to CARF 
(1980) Standard 3.4.3.1.1.1: 

Vocational evaluation services shall be provided on a systematic, 
organized basis . . . The range and scope of the evaluation 
services shall be^ sufficiently comprehensive to assess or obtain 
information concerning at least the following . . . 

d. interests, attitudes, knowledge of occupational information . . . 
(p. 29) • 

Botterbusch (1974) has provided an example of a cli"ent orientation: 

A. Client Orientation - 

Note to Evaluator: This orientation is to be oresented to each 
client before administering the STOUT U-BOLT ASSEMBLY WORK SAMPLE. 
The material does not have to be read verbatum. However, all in- 
formation should be covered. Deliver the presentation in an 
informal manner and pause to answer questions as necessary. The 
purpose of this orientation is (1) to inform the client about 
assembly and similar jobs which are related to this work sample 
and (2) to inform the client of specific traits or characteristics 
on which he is being evaluated. 



The STOUT U-BOLT ASSEMBLY WORK SAMPLE that you will be taking in a 
few minutes will help you and me to find out several things about 
you. One of the things it will tell us is how well you can do on 
assembly jobs and how well you like this type of work. Let me tell 
you a few things about assembly jobs. Many of the things we commonly 
use are put together in factories by people called '*assembl ers. " 
These people put together small or large parts to make things such as: 
television sets, radios, toys and dolls, and household items like 
mops, brooms, and picture frames. If you were employed in one of 
these assembly jobs, you would work inside a factory and be close to 
other people doing jobs like yours. You would probably sit at a 
bench and use small tools such as screwdrivers, drills, soldering 
guns, and wrenches to put things together. Often the parts would 
be brought to you in boxes or on a conveyer belt. In many assembly 
jobs, you would only complete one part of the finished product. 



You would pass this along to the next worker so he could finish his 
part. Your foreman would carefully supervise your work and other 
people called "inspectors" would check it carefully for mistakes. 
People don't have to have a lot of education to become assemblers, 
"but they have to be good with their hands, be able to do the same 
thing over and over again, enjoy working near people, and to fol- • 
low directions carefully. Sometimes people who do well in assembly 
jobs get promoted to more advanced jobs. Some may become foremen 
or inspectors themselves and others become assemblers who work on 
very complicated equipment. 



The evaluator then tells the client of nearby factories that hire 
assemblers or related jobs; what the wages are; what the working 
conditions are; and how many of these jobs are available. If no 
assembly jobs are available, give information about related jobs in 
the Handling Worker Trait Group (.887), such as packaging, dish- 
washers, cook helper, and custodians. 



This work sample will tell 'us things other than how well you like 
assembly work. It will help us to find out how well you can move 
your hands and how well you can use your fingers. After you try 
this work sample, both of us will know how long you can do the 
same thing without getting mad at the job, bored, or tired. 

Do you have any questions? 



If the client has no questions, the evaluator begins to read the 
instructions and demonstrate the work sample, (pp. 5-6) 

Th.is example points 'out the utility of a work sample orientation and 
emphasizes that it does not have to be a lengthy, cumbersome amount of in- 
formation in order to be effective. 



Administration 

One of the most basic areas of consideration that must be addressed when 
critiquing work samples is the clarity with which all the administration pro- 
cedures are spelled out. 

CARF (1980) has mandated "... work samples shall be , , . standardized 
as to materials, layout, instructions, and scoring" (p, 30). The vast majority 
of commercially developed work samples meet this guideline; however, it is not 
uncommon to find "in-house" work samples for which no standardized administra- 
tion procedures have been developed. This problem is important because a lack 
of standardization can contribute to erratic administration procedures which 
may have a profound effect on client performance. When this occurs, it is 
difficult to evaluate the extent to which client performance is representative 
of client skills and how much is the direct result of variations in the ad- 
ministration procedures. Lack of standardization also makes it very difficult 
to compare one client's performance to another's or to a norm group, since as 
was already discussed, .differences in administration procedures can influence 
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performance outcomes, Dunn (1977) has pointed out that even slight variations 
in the administration procedures can strongly affect performance. In an 
effort to deal with this problem, CARF (1980) has interpreted standardization 
to mean that all work samples should possess certain critical characteristics 
including: 

3.4.3.1.1.7.2b. Each work sample shall have an examiner's manual which 

specifies: (1) its relationship to the Dictionary of 
Occupational Titles , Occupational Divisions, Worker 
Trait Groups, or some appropriate job analysis system; 
(2) prerequisites; i.e., any specific work sample task 
requirements which might make administration of sample .. 
unfeasible for a given individual; (3) the work sample 
purpose; i.e., specifically what the sample is at- 
tempting to assess; (4) the materials and equipment 
used; (5) preoarations for testing and the layout of 
materials; (6; instructions to the individual; (7) 
instructions for timing, evaluating errors, and. scoring 
if applicable; (8) instructions for interpreting 
scores, (p. 79) 

It should be understood that work sample standardization is not, however, 
meant to imply that all work samples should be administered in all cases in a 
rigidly mechanical method that is blind to individual client needs and limi- 
tations. McCray (1979a) has pointed out the importance of instructional 
flexibility when working with clients whose learning styles are incompatible 
with the standardized instructional technique being offered. Perhaps this 
concept best exemplifies the notion that evaluators should have on hand 
standardized work samples that will be appropriate for the vast majority of 
clients being served, but when clients with special needs and limitations who 
cannot perform the standardized task at a level which intuitively appears to 
be representative of their general skills, the work sample and evaluator must 
be flexible enough so that the work sample can be adapted to better fit in- 
dividual client needs. For example, a standardized soldering work sample 
might use oral instructions during the learning phase, yet a particular client 
might be partially deaf. In such a case, the standardized administrative pro- 
cedure (once it was verified that it was inappropriate) should be able to be 
adapted (e.g., substitute written instructions, so that client performance is 
not adversely affected by rigidly standardized administration procedures). 
This concept is supported by CARF (1980) Standard 3.4.3.1.1.2 which states: 

Appropriate adaptive assessment tools and methods shall be used 
wherever possible with individuals having sensory, communication, 
or other functional impediments . . . which might invalidate 
otherwise standardized procedures (p. 30). 

It is imperative that the instructions to the evaluator in all areas in- 
cluding the information to be imparted to the client be clearly described. If 
procedures such as how many people the task can be given to at one time, client 
instructions, work sample conditions, or prerequisites are not easily under- 
stood, then this can lead to irregularities and errors in the day-to-day use 
of the work sample. Evaluators should carefully read, study, and in many 
cases self-administer a work sample in order to assess the clarity of in- 
structions. 



The setup, breakdown, layout, and construction of the work sample must be 
detailed so that tools, materials, etc., used in the work sample are consis- 
tently provided in a standardized method so as not to either positively or 
negatively affect client performance. Diagrams should parallel those used in 
industry, be clearly labeled, and spell out work sample dimensions. These 
factors are important for three reasons: (1) diagrams that don't parallel 
industry's can lead to making erroneous conclusions about client abilities, 
(2) a work sample may be ohysically incompatible with the space or layout 
requirements of the unit, and^(3) where norms or industrial standards are 
provided it is essential that the task performance methods and tools remain 
constant so that performance is not biased and the norms, therefore, rendered 
meaningless. For example, if a work sample manual specifies the work sample 
layout and MTM norms are going to be used, but the manual does not specify the 
exact physical dimensions of the 1 ayout (e.g. , position and distances between 
objects), then the MTM standards are useless since a variation in distance or 
position between two tools could dramatically improve or hinder nerformance 
capability. 

Nearly all psychological and performance oriented tests require that the 
testing conditions be clearly defined. This is because alterations in test 
conditions can have a profound impact on client performance. The same holds 
true for work samples. Evaluators should determine the extent to which the 
work sample conditions are defined, i.e., work environment, lighting, client 
dress, standing, sitting requirements. This is important for two basic rea- 
sons.. First, altering the work sample conditions could easily invalidate 
norms, reliability, and validity of a work sample. Second, depending on the 
uniqueness of the work sample conditions, it may be that the facility does 
not have the specified conditions available. For example, if a small engine 
repair work sample is to be administered in a noisy, dirty, industrial tyoe 
of environment yet the evaluation unit is headquartered in a sterile, quiet 
office, it may be impossible to adequately simulate the required conditions. 
In a case such as this, evaluators may decide that it would be inappropriate 
to use the work sample under conditions contrary to those specified in the 
manual . 

In many cases, the ease of administration of a work sample will be 
strongly affected by the estimated time needed for: (1) setup, breakdown, 
administration of instructions, etc., and (2) the time typically needed for 
the average client to perform the work sample. These factors may strongly 
influence the utility of the prospective work sample. For example, if a 
program typically has a high cl ient/evaluator ratio which necessarily reduces 
the amount of time an evaluator can spend with an individual client, then care 
should be taken to avoid selecting work samples which will require a great 
deal of one to one contact between the evaluator and the client. Similarly, 
if a program is set up so that the average client only spends one_or two days 
in evaluation, then the evaluator will often want to avoid selecting work 
samples that require a lot of time for the client to complete (e.g., six or 
seven hours). Thus,. it is important for the evaluator to be able to determine 
a relatively reliable estimate of how much administration/performance time 
will have to be devoted to the task for it to be usable. 

The final consideration relates to cost. This involves not only the cost 
of purchasing or constructing the entire work sample but alsothe approximate 
expense that can be expected per ad • ^'^istration of the work sample. Certainly, 
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overall cost is important, but administration costs can also be significant. 
Some work samples may be expensive to administer and this may impact on their 
practical day-to-day utility. In other cases, work samples with essentially 
similar purposes, but widely varying administration costs may be found. 
Thus, it is important for evaluators to keep these two cost factors in mind. 

Scoring 

It is imperative that all work samples have scoring procedures which are 
readily apparent and clearly defined. Scoring data generally include forms, 
tables, normative information, performance criteria for practice sessions, 
behavior observation checklists, readministration guidelines, quality cri- 
terion, and time limitations. It is important that scoring procedures be 
thoroughly outlined so that they can be reliably followed by all staff members 
expected to be using the work samples. 

The timing factors that are generally most pertinent to a work sample 
include: (l) whether the work sample is timed or untimed, (2) whether there 
is a flexible maximum time limit (e.g., if the work sample is not completed 
by the client within five hours it should be discontinued), (3) whether there 
is a definite time limit for completing performance (e.g., all clients will 
have 30 minutes to complete as much of the tasks as they can). In most 
cases, timing will involve recording how long it takes a client to complete 
the entire work sample task or discrete portions of the task. 

Quality factors generally are related to the errors that can be made on 
a work sample task. It is important that what constitutes an error or un- 
acceptable quality level be clearly described so that evaluation of a client's 
quality is as objective as possible. For example, on a measurement task, an 
error might be defined as any measurement in excess or less than one quarter 
inch of the desired length. In a case such as this, an error range has been 
described because although the measurement may not be entirely accurate, it 
is considered acceptable if it falls within the one quarter inch guidelines. 
Another example would be a single nut and bolt assembly wherein an assembly 
is defined as an error if the nut falls off the bolt when the assembly is 
lightly shaken. In this case, no error range has been defined, rather an 
absolute standard is provided. The assembly will either fall apart or it 
will not, so error determination is an easy process. Quality on some tasks 
may also be rated more subjectively. For example, a client's performance of 
a typing task with regard to overall neatness might be scored as above average, 
average, or below average, providing that a concrete definition of these terms 
is provided. In many cases, specific behavioral factors such as stamina, 
punctuality, cooperativeness, etc, will be scored by this method. In either 
case, it is essential that the scoring criteria, e.g., above average, far 
exceeds normal, etc, must be defined. 

Part C of CARF (1980) work sample standard 3.4.3.1.1.7.2 indicates that, 
. . c. ^competitive norms or industrial standards shall be established and 
used" (p. 30). Thus, when evaluators are reviewing new work sample manuals 
for possible inclusion within their program, they shoOld carefully examine 
the work sample manual in order to determine whether Competitive norms/ 
standards are available. Within this context, competitive standards denote 
the performance standards regularly applied or achieved by workers in industry. 
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It does not include sheltered workshop norms except in cases where the work 
sample was developed as a means of predicting workshop employment potential, 
and the worker group the client is expected to be competing against are 
clients. 

If a work sample does not have competitive performance standards available 
but does appear to still be useful, evaluators should carefully consider the 
feasibility of developing their own local norms. This may be a costly and 
time consuming process, since it may require the accumulation and analysis of 
much performance data as v^ell as the potential expense of having outside 
consultants (e.g., industrial engineers) develop the competitive standard. 
However, obtaining competitive production standards, particularly on work, 
samples directly related to specific jobs, can provide an important basis\for 
furthering one's understanding of a client's employment potential. ,\y 

Finally, most work sample manuals generally address behavioral factors 
which are to be rated based on client performance. It has already been pointed 
out that the rating levels must be clearly defined; however, it is imoerative 
that beyond this, definitions be provided of all behaviors that are to be 
rated. This is important because it helps increase inter-rater reliability, 
i.e., each rater has the same understanding and definition of each behavior^ 
on which clients are to be rated. This is critical, otherwise evaluators iriSy 
easily end up rating clients without any consistency because each evaluator 
has a different concept of the same behavioral term. For example, a bench 
work assembly work sample might be designed to address the behavior factor, 
stamina. The client is required to sit at a chair for three consecutive 
hours, assembling objects, and based on the client's behavior and feedback, 
the evaluator will be able to score the client's behavior with regard to doing 
sedentary work". - However , if the term stamina is not defined within the context 
of the work sample* task, it may be viewed differently by different evaluators. 
With regard to the aforementioned example, if no definition were provided, one 
evaluator might consider the term stamina to refer to the ability to work con- 
sistently, a second might regard it as ability to stay "on task," whereas a 
third evaluator may view it as the physical capacity to do sedentary work for 
extended periods up to three hours. Thus, it is important that if a work 
sample purports to assess certain behavioral factors, they be clearly defined 
and described, not just denoted. 



Learning Assessment 

Revel 1 and Wehman (1978) have separated the concepts of learning and per- 
formance by identifying two distinct phases of work sample testing. The 
learning phase is basically the period during which the client acquires the 
requisite skills and concepts needed to perform a task. The performance phase 
is the period following the learning phase in which the client is expected to 
demonstrate his task related competencies under formal testing conditions. 
Before individuals are requireci to perform tasks under testing conditions, 
they must have thoroughly learned and mastered all skills, concepts-, motions, 
etc., necessary to produce the expected behaviors. McCray (1979a) has carried 
this concept further by suggesting that in order to determine whether adequate 
learning has taken place, "work samples must have objective criteria which are 
measureable and define when adequate learning has taken place" (p. 9). Thus, 
work sample manuals should clearly specify what behaviors are to be learned by 
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tht. crnd of the learning phase, as well as concrete criteria which the evaluator 
can use as a basis for determining whether the client adequately understands 
what to do. For example, if at the end of the instructional phase of a nut and 
bolt sorting task the manual only specifies that "if the client has no ques- 
tions and appears to understand the task, proceed to performance testing," 
.then evaluators may easily assume that a client understands a task when in 
fact, he does not. If on the other Jnand, the manual specifies that "the 
evaluator should proceed to performance testing only after the client has 
correctly sorted ten consecutive nuts and bolts with 100% accuracy, without-^^^^^^^^ 
regard to speed," ev.aluators will have a concrete learning standard which" all 
clients must achieve prior to performance testing. The criteria defines what 
behaviors have to be mastered and the extent of mastery. Thus, the evaluator 
has a clearcut method for ascertaining whether a client appears to understand 
the task. Therefore, the likelihood of having unprepared clients who have not 
thoroughly understood the task instructions, erroneously proceed to the per- 
formance phase, is reduced. This also means the likelihood of misinterpret- 
ing a client's poor performance, as necessarily indicative of a lack of task 
related ability when, in fact, it may be due to inappropriate work sample 
instructional procedures, is reduced. Evaluators should study prospective 
work sample manuals to determine if they clearly spell out the tasks which 
a client must master and learn prior to being asked to perform the entire task. 
The availability of this information will help insure that accurate, reliable 
observations as to client capabilities take place. 



SUMMARY 

Choosing the best possible work sample to meet specific, well defined 
needs, is a challenging and difficult process. The purpose of this publication 
and the accompanying rating form found in the Appendix is to provide evaluators 
with guidelines which can help facilitate the selection process and, thereby, 
caution evaluators against randomly selecting and purchasing work samples. 
Many different aspects of the model work sample have been discussed; however, 
it should be remembered that few existing work samples will possess all the 
characteristics of the model. Therefore, evaluators must carefully weigh those 
factors they believe most relevant to their needs. In most cases, evaluators 
will find that there are three components which form the foundation for the 
viability of nearly all work samples: work samples should be appropriate to 
the needs and limitations of the clients to be served; they should relate to 
real jobs or training opportunities that actually exist in the available labor 
market, and their purpose(s) should be clearly defined. Work samples which 
fail to meet these basic, minimum criteria will frequently be of limited use- 
fulness and may, in fact contribute to a decline in the effectiveness of the 
evaluation service. On the other hand, work samples which soundly address 
these elements, as well as incorporate to varying degrees the other components 
of the model, will be exceptionally useful tools. 

The benefits of careful work sample selection are many and far reaching. 
Perhaps the most basic is the inevitable fact that clients and referral sources 
will be better served. Because the work samples used are carefully chosen to 
meet specific needs and objectives, observations as to client capabilities and 
limitations are likely to be more valid, more realistic, and serve as a more 
effective basis for decision making. In addition to this, money and personnel 
will not be wasted on purchasing or developing irrelevant, unreliable work 
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samples of questionable utility which at best, may simply sit unused, taking 
up valuable space, or which at worst, may contribute to erroneous interpre- 
tations as to client capabilities. The long-term effect of careful work sample 
selection is that an increased number of clients will be successfully re- 
habilitated as a direct result of an effective and efficient vocational 
evaluation service. 
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APPENDIX 



Suggested Guidelines for Evaluating Work Samples 

Work Sample (w,sO Title: 

Purpose for which w.s. is being reviewed: 



Comments : 



Item 



APPROPRIATENESS TO CLIENT POPULATION 

a. Prerequisites realistic in view of client 
abilities 

b. Job areas assessed realistic in view of 
client abilities 

c. Instructional mode realistic or could be 
adapted to fit client abilities 

d. Instructional mode parallels that used 
in industry 

WORK SAMPLE PURPOSE 

a. Purpose(s) clearly stated and defined 
Purpose(s) appropriate to program goals 
Purpose does not duplicate existing w»s. 



Purpose does riot duplicate other assessment 
tools 

Compliments existing w.s. and other 
assessment tool s 



Rating 
Yes No 



□ □ 



□ □ 



Item 



Rating 



Yes 



3. RELATIONSHIP TO AVAILABLE LABOR/TRAINING MARKET. 

a. Directly related to actual jobs in available 
labor market 

b. Directly related to actual training programs 
in available labor market 

c. Extent of representativeness/validity 
documented 

d. Job analysis or similar information available 

e. Face validity apparent 

4. ORIENTATION- 

a. Standardized orientation present 

Orientation is job related and understandable 
to clients 

c. Sufficient amount of information to relate 
w,s. to available labor market 

5. ADMINISTRATION 

a. Instructions to evaluator standardized and 
understandable 

b. Instructions to client standardized and 

understandable 

c. Materials used clearly described 

d. Equipment used clearly described 

e. Layout, setup and breakdown, and 
construction clearly described 

f. Diagrams understandable and comparable to 
those used in industry 

g. Safety precautions described 

h. Time necessary for evaluator to administer 
and score is realistic for program 
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Item 



Rating 



Yes 



No 



5. ADMINISTRATION (cont.) 



1. Time necessary for average client to complete 
w.s. is realistic for program 

j. Per administration costs described 

k. Purchase/development costs described 

SCORING 

a. Timing factors thoroughly described 

Quality/error factors thoroughly described 



b. 
c. 



Competitive or industrial norms available 
and defined 



d. Behavior rating factors thoroughly described 
7. 'LEARNING ASSESSMENT 

a. Mastery criteria for learning phase provided 

b. Mastery criteria realistic in terms of ensuring 
adequate client learning precedes performance 
testing 

TOTAL 



□ 



nr 



] $ 



$ 



8. SUMMARY COMMENTS (rationale for selecting/rejecting w.s.) 
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MDC ORDER FORM 

This form is for use when ordering MDC publications. For additional information on 
available audiovisual programs, contact the MDC Information Service. 

Quantity Title and Cost Total 

1. Vocational Evaluation and Work Adjustment Services in 

Vocational Rehabilitation - $2.00 

2. Observation and Client Evaluation in Workshops: 

A Guide and A Manual - $1.50 

3. Road to New Horizons: Adjustment Training - $2.00 

4. MDC Behavior Identification Format - $2.00 

5. Effective Report Writing in Vocational Evaluation and 

Work Adjustment Programs - $2.50 

6. Individualized Client Planning for Work Adjustment 

Services - $1.50 

7, Adjustment Services in Rehabilitation: Emphasis on 

Human Change - $3.00 

8. Client Rating Instruments for Use in Vocational 

Rehabilitation Agencies - $1.50 

9. Work Sample Evaluation of Blind Clients: Criteria 

for Administration and Development - $1.50 

10. The Use of Psychological Tests with Individuals Who 

are Severely Disabled - $2.00 

11. Handbook for Analyzing Jobs - $3.00 

12. Vocational Evaluation Project Final Report - $2.00 

13. Work Evaluation and Adjustment: An Annotated 

Bibliography, 1947-1977 - $3.00 

14. Work Evaluation and Adjustment: An Annotated 

Bibliography, 1978 Supplement - $1.50 j 

15. Work Evaluation and Adjustment: An Annotated 

Bibliography, 1979 Supplement - $1.50 

16. A Structured Guide for Selecting Training 

Materials in Adjustment Services - $1.50 

17. Training Programs in Work Evaluation, Adjustment and Facility 

Management (Single copies free - multiple copies 25i each) 

___ 18. Work Sample Manual Format - $2.00 

19, VEWAA-CARF Vocational Evaluation and Work Adjustment Standards 

with Interpretive Guidelines and VEWAA Glossary - $1.00 

20. A Guide to Job Site Evaluation - $1.50 

21. Occupational Information in Vocational Evaluation - $1.00 ~ 

22. A Scale of Employability for Handicapped Persons - $2.00 

23. The Individual Evaluation Plan - $1.50 

24. Instructional Materials in Independent Living - $1.50 

_____ 25. Pennsylvania Model Individualized Written Program Plan for 

Vocational Rehabilitation Facilities - $2.00 

O 
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Quantity Title and Cost Total 

26. Psychological Testing in Vocational Evaluation - $2.00 

Zl. Client Orientation: A Key to Successful Involvement 

in Rehabilitation Facility Services - 25(t 

28. Learning Assessment in Vocational Evaluation - $1.00 

29. 'Job Readiness Training Curriculum - $2.50 

30, An Interpretation of VEWAA/CARF Work Sample Standards - $1.00 

31. Independent Living Evaluation-Training Program - $2.50 

32. Production Improvement in a Rehabilitation Workshop - $2.50 

33. Goal Attainment Scaling in Rehabilitation - $2.00 

34. Gathering Information for Evaluation Planning - $2.00 

35. The Dictionary of Occupational Titles in Vocational 

Assessment: A Self-Study Manual - $3.00 

• 36. An Activities of Daily Living Curriculum for Handicapped 

Adults - $15.00 

37. Workshop Production Management: Motion and 

Time Study - $2.00 

38. A Comparison of Commercial Vocational Evaluation 

Systems - $5.00 

39. Vocational Evaluation and Work Adjustment Bulletin - 48 back 

issues, dating from 1968 to present, are available at $3.00 
per issue. Four issues are published yearly--March, June, 
September and December. Indicate issue(s) and year(s) 

here: 

40. Developing Purpose Statements for Rehabilitation 

Facilities - $2.50 



PLEASE PRINT OR TYPE 
Ship to: 

Name: 

Organization: 



Address: 

City: 

State: Zip: 

Phone: 



Total 



Mail to: Materials Development Center' 

Stout Vocational Rehabilitation 

Institute 
University of Wisconsin-Stout 
Menomonie, Wisconsin , 54751 



Payment enclosed 



Purchase order attached 
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